A Bayesian evaluation framework for subjectively annotated visual recognition tasks

An interesting development in automatic visual recognition has been the emergence of tasks where it is not possible to assign objective labels images, yet still feasible collect annotations that reflect human judgements about them. Machine learning-based predictors for these rely on supervised training models behavior annotators, i.e., what would average person's judgement be an image? A key open question this type work, especially applications inconsistency with can lead ethical lapses, how evaluate epistemic uncertainty trained predictors, comes from predictor's model. We propose a Bayesian framework evaluating black box regime, agnostic internal structure. The specifies estimate predictor respect by approximating conditional distribution and producing credible interval predictions their measures performance. successfully applied four image classification use subjective judgements: facial beauty assessment, social attribute assignment, apparent age estimation, ambiguous scene labeling.